Copy-resistant symbol having a substrate and a machine-readable symbol instantiated on the substrate

ABSTRACT

A variety of forms of machine-readable symbols are disclosed, as well as methods and systems of constructing machine-readable symbols, methods and systems of acquiring machine-readable symbols, and methods and systems of decoding machine-readable symbols.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a nonprovisional of, and claims the benefit of thefiling date of, each of the following provisional applications: U.S.Prov. Pat. Appl. No. 61/384,579, entitled “METHODS AND SYSTEMS TO MAKE,IMAGE AND PROCESS BARCODES AND OTHER MACHINE READABLE DATA,” filed Sep.20, 2010 by Robert K. Rowe, Alex Litz, and Ryan Martin; U.S. Prov. Pat.Appl. No. 61/392,874, entitled “OPTICAL MULTIPLEXING FOR BARCODEACQUISITION,” filed Oct. 13, 2010 by Robert K. Rowe and Ryan Martin;U.S. Prov. Pat. Appl. 61/407,840, entitled “METHOD OF STATISTICALINTERPRETATION OF BARCODE IMAGES,” filed Oct. 28, 2010 by Robert K.Rowe; and U.S. Prov. Pat. Appl. No. 61/429,977, entitled “COPY-RESISTANTTOKENS,” filed Jan. 5, 2011 by Robert K. Rowe. The entire disclosure ofeach of these provisional applications is incorporated herein byreference in their entireties.

This application is related to the following concurrently filed,commonly assigned applications: U.S. patent application Ser. No.13/236,953, entitled “MACHINE-READABLE SYMBOLS” by Robert K. Rowe et al;U.S. patent application Ser. No. 13,237,013, entitled “MACHINE-READABLESYMBOLS” by Robert K. Rowe; and U.S. patent application Ser. No.13/237,074, entitled “MACHINE-READABLE SYMBOLS” by Robert K. Rowe.

BACKGROUND OF THE INVENTION

This application relates generally to machine-readable symbols. Morespecifically, this application relates to methods and systems forfabricating, acquiring, and processing machine-readable symbols.

Since their origins in the late 1940's, barcodes and other types ofmachine-readable symbols have become ubiquitous. They are used in a widerange of applications to identify items in a way that may be understoodby a variety of devices. Perhaps the most common example is the use ofbarcodes to identify retail products with the Global Trade Item Numbers(“GTIN”) or Universal Product Code (“UPC”) symbologies. These systemsare examples where machine-readable symbols are used to identifygenerally fungible products for sale, with information encoded in thebarcode to identify characteristics of products being sold, includingsuch information as an item number, a weight of the product, a price forthe product, and the like. Other barcode symbology uses that identifyclasses of products are implemented in any number of inventory-basedsystems, such as in factories that use barcodes to track componentsupplies and to automate reordering when supplies of certain componentsare near depletion.

Other types of systems assign unique barcodes to items rather thanassigning barcodes to groups of items. One of the more important ofthese is the GS1 supply-chain system, which implements a series ofstandards that are designed to improve supply-chain management. Incombination with other standards, barcode standards are promulgated inthis system to allow unique identification of products in manufacturingand other contexts. The Air Transport Association (“AITA”) implements asystem of barcodes on aircraft boarding passes, a system that is tied tosecurity and safety applications, and the use of barcodes in managingaccess to entertainment events have also become increasingly widespread.

Barcodes are also used for unique identification of living beings,notably in biological research in which animals are tagged with barcodesto track individual the behavior of individual animals, particularly inlarge-population environments where individual identification of theanimals is otherwise difficult (such as for the tracking of behavior ofbees in hives). Barcodes have also been used for the identification ofhuman beings, such as in medical environments where wristbands havingsymbols that encode patient information are deployed.

While many deployments of machine-readable symbols are effected byattaching labels to items with printed barcodes, there are otherimplementations in which the symbols are incorporated directly onto thepart being marked. This may be accomplished by such techniques as laseretching, chemical etching, dot peening, casting, machining, and otheroperations, and is particularly common in supply-chain applications.

The very ubiquity of machine-readable symbols means that there are manydifferent circumstances in which the symbols may be difficult to readreliably: this may be because, among other reasons, the symbol itself isof poor quality; because the shape, color, or configuration of an objecton which it is instantiated presents imaging challenges; or theenvironment in which it is to be read presents challenges. While anumber of processing techniques have been developed to address suchdifficulties, many of these remain ineffective under a variety ofconditions so that a need remains in the art for improved acquisitiontechniques.

In addition, many applications for machine-readable symbols introducethe risk of a variety of types of fraud. Software is widely available,both on the Internet and through other commercial avenues, that allowindividuals to generate barcode symbologies that may be improperlyaffixed to items. Fraud can also be committed by copying barcodes andinappropriately attaching them to items so that the items aredeliberately misidentified. Such copying is, moreover, not limited tothe copying of barcodes to be attached to items but can also becommitted with direct-part marks that are incorporated directly on itemsby examining and reproducing the marks improperly. Such fraud can notonly have significant financial consequences, but can also have theeffect of interfering with supply-chain monitoring and scenarios caneven be envisaged in which such copying is used to commit batteries andother physical crimes against individuals through the deliberatemislabeling of medications, medical parts, and even the patient himself.There is accordingly also a need in the art to enhance the security ofmachine-readable symbols.

SUMMARY

Embodiments of the invention are directed to a variety of forms ofmachine-readable symbols, to methods and systems of constructingmachine-readable symbols, to methods and systems of acquiringmachine-readable symbols, and to methods and systems of decodingmachine-readable symbols.

In a first set of embodiments, methods and systems are provided foracquiring an image of a machine-readable symbol. The machine-readablesymbol is illuminated with a plurality of illumination sources disposedrelative to the machine-readable symbol to define a plurality ofdistinct illumination geometries. For each illumination geometry, arespective raw image of the machine-readable symbol is obtained. Atleast one of the respective raw images includes a dark region.Information from the respective raw images is combined to generate asingle image of the machine-readable symbol.

In some of these embodiments, the machine-readable symbol comprises aprinted barcode, but may take other forms in alternative embodiments.

There are various ways in which information from the respective rawimages may be combined. For example, a nonminimum pixel may be selectedfrom each of the respective raw images. A bilateral filter may beapplied to at least one of the respective raw images. A pixel intensitymay be averaged across the respective raw images. Information from therespective raw images may be uniformly or nonuniformly weighted indifferent embodiments, such as by applying a nonuniform weighting inaccordance with a determination of a quality for each of the respectiveraw images. An intrinsic characteristic of on object on which themachine-readable symbol is instantiated may be estimated, such as byprocessing the raw images with a photometric stereo technique to derivea measure of surface topography and reflectance of the object.

In certain embodiments, the illumination sources define a balancedarrangement in which illumination portions of the respective raw imagesvary between illumination conditions in a complementary fashion.

In a second set of embodiments, methods and systems are also provided ofacquiring an image of a machine-readable symbol. The machine-readablesymbol is illuminated with a plurality of illumination sources havingdifferent illumination spectra. An image of the illuminationmachine-readable symbol is collected, and chromatic components of thecollected image are separated.

The machine-readable symbol may comprise a printed barcode in someembodiments, but may take other forms also.

The plurality of illumination sources may be disposed relative to themachine-readable symbol to provide a plurality of distinct illuminationgeometries, such as by having at least two of the illumination sourcesdisposed at different azimuthal and/or elevation angles relative to themachine-readable symbol.

In some instances, illumination from at least one of the illuminationsources is polarized so that the method further comprises separatingpolarization components of the collected image.

The illumination sources may also take different forms in differentembodiments. In one embodiment, at least one of the illumination sourcesprovides diffuse illumination, while in another embodiment, at least oneof the illumination sources provides substantially directionalillumination. The illumination sources might also provide a plurality ofillumination wavelengths, but with different illumination sourcesproviding the illumination wavelengths at different relativeintensities. In another embodiment, each illumination source is insteadsubstantially monochromatic.

A third set of embodiments provides a handheld device and method ofacquiring an image of a machine-readable symbol with the handhelddevice. A plurality of raw images of the machine-readable symbol takenwith the handheld device are received. The plurality of raw images areregistered, and information from the registered images is combined togenerate a single image of the machine-readable symbol.

In some of these embodiments, the machine-readable symbol comprises aprinted barcode, while in others it comprises a direct-part mark. Thevarious types of combinations of information from the images may beperformed in various embodiments. The handheld device may comprise amobile telephone or a tablet computer, among others.

A fourth set of embodiments provides a multi-mode machine-readablesymbol. A first machine-readable symbol is instantiated on an object ata first location and is readable by a first methodology. A secondmachine-readable symbol is instantiated on the object at a secondlocation that overlaps the first location and is readable by a secondmethodology different from the first methodology.

In some instances, a third machine-readable symbol is instantiated at athird location that overlaps the first and second locations and isreadable by a third methodology different from the first and secondmethodologies.

In specific embodiments, the second machine-readable symbol comprises apattern of marks formed in a surface of the object. The firstmachine-readable symbol may comprise a barcode printed on a surface ofthe object or on a conformal layer applied over the surface of theobject.

In a fifth set of embodiments, methods and systems are provided fordecoding a machine-readable symbol configured as a set of marks form ina surface of an object. A presence or absence of a mark in each cell ofan array of cells designates a binary state of the cell. An image of themachine-readable symbol is acquired. Cells of the array are identifiedfrom the acquired image. Reference cells of the array are evaluated inaccordance with a reference standard for the machine-readable symbol toidentify optical characteristics consistent with the presence or absenceof a mark in the reference cells. Nonreference cells of the array areclassified in accordance with the identified optical characteristics todetermine the presence or absence of a mark in the nonreference cells.Classifications of the nonreference cells are compiled into a binarygrid, which may then be decoded.

In some embodiments, the image of the machine-readable symbol may beacquired by illuminating the machine-readable symbol with a plurality ofillumination sources having different illuminating spectra, with thereference cells being evaluated and the nonreference cells beingclassified by separating chromatic components of the acquired image. Insome cases, the illumination sources are disposed relative to themachine-readable symbol to provide a plurality of distinct illuminationgeometries, such as by having different azimuthal or elevation angles.The illumination sources may also provide diffuse or directionalillumination, and may be substantially monochromatic.

The reference cells may be evaluated and the nonreference cellsclassified by determining a statistical measure of pixel values withinthe cells, such as a mean or standard deviation of the pixel values.

In a sixth set of embodiments, a copy-resistant symbol is provided as amachine-readable symbol instantiated on a substrate. Themachine-readable symbol represents a combination of substantive andencrypted security information. A decryption of the encrypted securityinformation identifies a security feature of the copy-resistant symbolidentifiable through optical imaging of the copy-resistant symbol.

The substrate or the machine-readable symbol may comprise an opticallyvariable material, and the machine-readable symbol may be printed overthe substrate or incorporated within the substrate. The security featuremay comprise an identifying mark comprised by the substrate or comprisedby the machine-readable symbol. For example, the security feature maycomprise an angular relationship of the identifying mark and a referencecomprised by the copy-resistant symbol or may comprise a spatialrelationship between the identifying mark and a reference comprised bythe copy-resistant symbol.

The encrypted security information may be encrypted according to asymmetric or asymmetric encryption algorithm. In some embodiments, thecopy-resistant symbol further comprises a supplementary layer distinctfrom the substrate and the machine-readable symbol, with the securityfeature comprising an identifying mark comprised by the supplementarylayer.

Such copy-resistant symbols may be read by optically acquiring themachine-readable symbol from the copy-resistant symbol and decoding itto derive a message. A portion of the message may be decrypted byapplying a decryption key, and determining a security feature from thedecrypted portion of the message. Physical presence of the securityfeature on the copy-resistant symbol may then be confirmed.

BRIEF DESCRIPTION OF THE DRAWINGS

A further understanding of the nature and advantages of the presentinvention may be realized by reference to the remaining portions of thespecification and the drawings, wherein like reference labels are usedthrough the several drawings to refer to similar components. In someinstances, reference labels are followed with a hyphenated sublabel;reference to only the primary portion of the label is intended to refercollectively to all reference labels that have the same primary labelbut different sublabels.

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawings will be provided by the Office upon request and paymentof the necessary fee.

FIG. 1 is a schematic illustration of one type of optical reader thatmay be used in embodiments of the invention for machine reading ofsymbols;

FIG. 2 provides a perspective view of an optical reader having threesources of illumination (COLOR);

FIG. 3 illustrates the use of multiple monochromatic imagers combinedwith chromatic beam splitters;

FIG. 4 illustrates the use of nominally monochromatic illuminators indifferent illumination geometries (COLOR);

FIG. 5 shows the structure of a typical Bayer color-filter array withoverlapping passbands (COLOR);

FIG. 6A is a schematic illustration of one example of a mobilecommunications device with the invention may be embodied;

FIG. 6B is a schematic illustration of an internal structure of a mobilecommunications device with which the invention may be embodied;

FIG. 6C is a flow diagram summarizing methods of using a mobilecommunications device to acquire an image of a machine-readable symbol.

FIG. 7A shows a barcode imaged by using illumination at eight differentazimuthal angles;

FIG. 7B shows images generated from the images of FIG. 7A afterundergoing pixelwise sorting and redisplay;

FIG. 8 demonstrates the effect of applying balanced acquisition tobarcode images;

FIG. 9 shows estimated illumination profiles for the images of FIG. 7A,generated using a wavelet smoothing function applied to the images inFIG. 7A;

FIG. 10 shows reflectance images generated from raw images in FIG. 7Aand the estimated illumination profiles of FIG. 9;

FIG. 11 shows log(reflectance) or pseudo-absorbance generated from theimages shown in FIG. 10;

FIG. 12 shows rescaled raw-intensity and pseudo-absorbance images forthe upper left images in FIGS. 5 and 11;

FIG. 13 provides an illustration of components of a dual-mode barcode;

FIG. 14 shows red, green, and blue color planes from a dot-peenedmachine-readable symbol;

FIG. 15 is a color image generated from the three raw color planes shownin FIG. 14;

FIG. 16 shows a dot-peened machine-readable symbol with cell locationsestablished (COLOR);

FIG. 17 is a close-up of a portion of FIG. 16 that shows that themanifestation of diffuse (red) illumination and two different direct(blue, green) illumination are all different from each other (COLOR);

FIG. 18 is an image of a dot-peened machine-readable symbol showing adiscrimination of reference cells either containing or not containing adot peen (COLOR);

FIG. 19 shows the results of applying a classification method to unknowncells for a dot-peened machine-readable symbol (COLOR);

FIG. 20 shows red, green, and blue color planes from an industrialbarcode;

FIG. 21 is a color image generated from the three raw color planes shownin FIG. 20;

FIG. 22 is a close-up of the central portion of the barcode shown inFIG. 21 (COLOR);

FIG. 23 shows reference cells of the industrial barcode of FIG. 20 usedfor subsequent classification (COLOR);

FIG. 24 shows the results of applying a statistical classificationtechnique to the industrial barcode of FIG. 20 (COLOR);

FIG. 25 is a flow diagram that summarizes methods of the invention forreading copy-resistant machine-readable symbols in accordance with someembodiments;

FIG. 26 provides a schematic illustration of how information may beencoded within a machine-readable symbol taking the form of atwo-dimensional barcode;

FIG. 27A provides an example of a holographic substrate over which anonholographic machine-readable symbol may be printed as illustrated inFIG. 27B (COLOR);

FIG. 28 provides an example of a symbology that uses a holographicmachine-readable symbol printed over a nonholographic substrate (COLOR);and

FIGS. 29A and 29B provide examples of spatial relationships betweenelements of a copy-resistant machine-readable symbol that may be used asfeatures according to embodiments of the invention (COLOR).

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

1. Introduction

Embodiments of the invention are directed to “machine-readable symbols,”which are symbols that comprise one or more marks capable of beingacquired by an imaging system, with the resulting image interpreted by acomputational system. Barcodes are examples of machine-readable symbols,and while references are sometimes made specifically to barcodes in thisdisclosure for purposes of illustration, it is to be understood thatembodiments of the invention are relevant to any type ofmachine-readable symbol. Furthermore, in implementations where barcodesare used, the invention is not limited by the symbology used ingenerating the barcodes and may accommodate any symbology. Examples ofsuch symbologies include one-dimensional symbologies such as Codabar,Code 11, Code 128, Code 32, Code 39, Code 93, EAN-13, EAN-8, EAN-99,EAN-Velocity, Industrial 2 of 5, Interleaved 2 of %, ISBN, UPC-A, UPC-E,and other symbologies. Further examples include two-dimensionalsymbologies such as Aztec Code, Code 16K, PDF417, Compact PDF417, MicroPDF417, Macro PDF417, DataMatrix, QR Code, Semacode, and other formats.The invention may also accommodate both monochromatic and color barcodesymbologies, including, for example, the High Capacity Color Barcode(“HCCB”) symbology. The inclusion of color in barcode symbology is oneexample of a more general class of multidimensional barcodes that encodeinformation using nonspatial dimensions, and other such multidimensionalbarcodes that use nonspatial dimensions are also accommodated inembodiments of the invention.

Other examples of machine-readable symbols that may be used inembodiments of the invention include machine-readable text,human-readable text that is amenable to optical character recognition(“OCR”) techniques for its machine interpretation, magnetic-inkcharacters written according to the magnetic-ink character recognition(“MICR”) format described in the International Organization forStandardization (“ISO”) publication ISO 1004:1995, and other similartypes of symbols.

The machine-readable symbol is generally instantiated on a substrate,although the form of instantiation may take a variety of forms indifferent embodiments, frequently depending on the material of thesubstrate and on the manner in which the symbol is to be associated withan item. Consider those instances where the symbol is to be associatedwith an item by incorporating the symbol directly on the item. If theitem comprises a surface on which ink will adhere, the symbol may beprinted on the item using any printing techniques conventional for suchsurfaces. Alternatively, the symbol may be incorporated as part of theitem by modification of a surface of the item using such techniques aslaser etching, chemical etching, dot peening, casting, machining, andthe like. Such techniques may also be used when the symbol is associatedwith the item through the use of a separate label, which may be affixedor attached to the item, or which may in some embodiments be positioneddistinct from the item in a way manner that makes the association clear.An example of such an instance would be a barcode label affixed to ashelf on which the item rests.

The combination of substrate and machine-readable symbol may further beaccompanied by a supplementary layer, examples of which includelaminate, coating, or other covering layers that are generallycoextensive with the symbol although layers that are not coextensive mayalso be provided in some embodiments.

Conventional ways of acquiring information from machine-readable symbolsbegin by acquiring a digital image of an area of the object containingthe symbol using an imager, commonly a complementarymetal-oxide-semiconductor (“CMOS”) imager or a charge-coupled device(“CCD”). In some cases, auxiliary lighting is also used to illuminatethe object during image acquisition.

Generally, the digital image of the symbol is then processed byspecialized software that performs multiple tasks. First, the softwareanalyzes the image to determine whether a machine-readable symbol ispresent or not. If present, the software determines the exact locationand orientation of the symbol. The software may then apply variousspatial transformations, image enhancements and other processing to thesymbol, after which the individual elements of the symbol (which areusually binary) are extracted and decoded. Although many symbols haveerror correction built into them, the techniques may still be unable toprocess and decode the symbol because the overall image quality is poor.

One way of addressing poor-quality symbol images uses acquisition ofmultiple images under multiple optical conditions, particularly throughthe use of multiple illumination angles and/or imaging angles. Each ofthe images may then be analyzed to find one that is sufficiently good todecode. But in some situations, one example of which occurs when thesymbol is instantiated on a round object, no one illumination conditionis sufficiently able to illuminate the entire symbol so that no oneimage of the symbol is adequate for decoding.

Another variant uses acquisition of two images of a machine-readablesymbol illuminated or imaged from different angles, in which the areasof specular reflection change between differently illuminated images.This technique attempts to accommodate for the fact that the specularreflection may obscure the local symbol features. The resulting imagesare then combined by performing a pixel-by-pixel minimum operation.While such a technique may produce a readable image where eachindividual symbol image is unreadable, the procedure requires bothimages to be adequately illuminated over the entire region so that a“minimum” operation applied to a dark region on one of the two images soprocessed will cause a dark region in the resulting image.

A further variation relies on collection of multiple symbol images underdifferent illumination or imaging conditions so that a subset of thesymbol images may be extracted and stitched together in an attempt toderive a single readable symbol. This method requires the symbolposition to be properly determined so that contiguous subsets of symbolfeatures may be extracted, processed, and analyzed to determine theirreadability and to combine the readable subsets into a stitched barcodethat is then hopefully readable in its entirety. Processing ofindividual symbol elements is computationally intensive and specializedfor each type of barcode or other machine-readable symbol encountered.As with the previous method, only a single raw image is used to defineeach point or barcode element of the composite image, which makes themethod sensitive to cases where no one raw image has adequate signal incertain portions of the symbol.

Multiple images of a machine-readable symbol may also be collected undermultiple imaging conditions by using a single imager to collect asequence of images, each illuminated under a different condition, suchas with different illumination angles. Such sequential collection may beproblematic for rapidly moving objects. If it is possible to align suchimages at all, the required image processing may be quite complicated,time-consuming, and error-prone. Alternatively, multiple simultaneousimages may be acquired using multiple imagers with different imagingcharacteristics, such as different imaging angles. Such configurationsmay be large and costly due to the multiple imaging subsystems. Further,such systems are limited to a single set of illumination conditions thatare used to illuminate the object when the multiple images are soacquired.

Multiple images of a machine-readable symbol may be collectedsimultaneously under multiple illumination conditions using a singlecolor imager that is used to collect images due to the multipleillumination conditions. Each illumination condition corresponds to adistinct monochromatic wavelength that, in turn, corresponds to adistinct color channel. As such, such a prior-art technique teachesestablishing a one-to-one correspondence between illumination geometryand a single color channel.

In addition to the term “machine-readable symbol” provided above,certain terminology is used herein in accordance with precisedefinitions.

“Azimuth angle” refers to the angle in a plane perpendicular to animaging axis. The azimuth angle is measured from some defined point or,if not stated, the measurement point may be arbitrary, but the relativeazimuth angles between two or more points may be operative.

“Elevation angle” refers to the angle in a plane containing an imagingaxis. This angle is measured from the imaging axis. The 180° ambiguityassociated with the measurement may be obvious from context or may befurther defined if critical to the specific reference.

“Hologram” refers to a conventional hologram of all types, includingwhite-light holograms. The term also refers to other material andmaterial structures that change optical properties as a function ofillumination or imaging angles. For example, a “hologram” includesoptically variable inks that change color as a function of illuminationand/or imaging angle. The term also encompasses a variety of diffractiveand refractive structures that change spectral characteristics as afunction of illumination and/or imaging angle. The term also encompassespolarization-sensitive material whole reflectivity changes as a functionof the polarization angle of the light incident on it and/or thepolarization angle through which the material is viewed.

“Illumination geometry” refers to the spatial characteristics of theillumination light. In a case where the light is highly directional,“illumination geometry” refers to the elevation and azimuth angles of anillumination beam. Such light might have an elevation angle near zerodegrees (or 180°), which is also referred to sometimes as being“on-axis.” Such directional light might also have an elevationsubstantially different than zero degrees (or 180°), in which case theazimuth angle is also used to specify the illumination geometry.Directional light may be collimated such that all illumination anglesacross the beam profile are substantially the same. Alternatively, thedirectional light may be converging or diverging relative to the axis ofillumination. Further, the directional light may comprise some weightedcombination of collimated, converging, and diverging beams with the sameor different illumination angles. Alternatively, the light may benondirectional or diffuse, with no preferred directionality. The lightmay have some complex relationship between illumination directions,spatial distributions, and intensities. In the case of broadband ormultiband (i.e., multiple different discrete wavelength) illuminationlight, the relationship between illumination directions, spatialdistributions, and intensities may also vary as a function ofwavelength. In some cases, such as with transparent or translucentobjects, lighting may be provided on the side of the object opposite theside that is imaged. All such characteristics of light also fall withinthe scope of the term “illumination geometry,” as does any mixture ofdirectional and diffuse lighting.

A “multi-imaging” sensor refers to a sensor that comprises at least asingle imager and a mechanism to collect a plurality of images of anobject during a single measurement session under different opticalconditions. The different optical conditions may be differentillumination angles, different degrees of illumination diffusivity(i.e., directional lighting versus diffuse light), differentillumination wavelengths, different illumination polarizationconditions, different imaging angles, different imaging polarizations,different imaging exposure times, and other such differences of thesort. The imager or imagers may be monochromatic or they may incorporatecolor filter arrays or other known mechanisms to collectdifferent-wavelength images simultaneously. Thus, within the scope ofthe present invention, a plurality of images may be collectedsimultaneously using color filter arrays and other such multiplexingtechniques.

“Illumination spectrum” refers to the wavelength characteristics and/orother characteristics of the light used to illuminate the object. Thedistribution of wavelengths in a particular illumination spectrum maycomprise a plurality of distinct wavelengths or a continuum ofwavelengths. The illumination spectra from two different illuminationsources are said to be distinct if the relative intensities of thewavelengths in the first illumination spectrum are measurably differentthan the relative intensities in the second illumination spectrum. Thus,two illumination spectra may be distinct even though some portion (orall) of the same wavelengths of light are present in the two spectra.Similarly, if other optical characteristics of two illumination spectraare measurably different (e.g., polarization, spatial distribution,angular distribution, etc.), the two spectra are also said to bedistinct whether or not the relative intensities of the illuminationwavelengths are the same.

An “optically multiplexed” sensor refers to a multi-imaging sensor thatis able to simultaneously collect a plurality of images of an objectduring a single measurement session under different optical conditionsby coupling two optical parameters in some way. For example, differentillumination geometries may be coupled with different illuminationspectra such that a color imager (which intrinsically distinguishesbetween different spectra) is able to be used as a mechanism fordistinguishing between different illumination geometries. In a similarway, polarization may be coupled with illumination spectra to enable thedistinction between different polarization conditions based on differentspectra characteristics. Also within the scope of the present invention,a polarization-sensitive imager may be used for optical multiplexing bycoupling another characteristic such as illumination spectra,illumination geometry, etc., to polarization state. Other similarcoupling and detection of optical characteristics also fall within thescope of the present invention.

“Optically variable” material refers to a material that responds tochanges in illumination characteristics with a measurable change in anoptical property. Examples of optically variable materials thus includeholographic materials, materials that incorporate diffraction gratings,optically variable inks and pigments, iridescent coatings, and the like.

A “security feature” refers to an identifying marking and may take theform of words, symbols, patterns, colors, textures, fiducial marks,surface finishes, and the like. In some instances, a security feature isunique to an item associated with a machine-readable symbol. This mayparticularly be the case when the substrate is comprised by the item sothat the identifying marking is produced on the item as a consequence ofrandom outcomes in the manufacturing process used to make the item. Buteven when the manufacturing process has predictable outcomes inproducing features, such features may be used as a security feature.

2. Acquisition

a. Optically Multiplexed Acquisition

Machine-readable symbols of the types defined above may be read by anumber of different imaging devices according to different embodimentsof the invention. One such device is illustrated schematically in FIG.1.

The imaging device 100 shown in FIG. 1 includes an optical window 102through which images of the machine-readable symbol may be collected forimaging by a digital imaging system 118. In this configuration, thefocal plane of the imaging system 118 is substantially coincident withthe exterior surface of the optical window 102. In some embodiments, theoptical window 102 may be omitted or the position of the window could bedisplaced from the focal plane. Examples of devices in which the opticalwindow is omitted are provided with FIGS. 2 and 4 below. Illuminationsystems 110 may be provided as part of a mechanism for collecting theimages. In the illustrated embodiment, the illumination systems 110comprise light sources 108 and optics that interact with the optics ofthe digital imaging system 118. The number of illumination sources 108may conveniently be selected to achieve certain levels of illumination,to meet packaging requirements, and to meet other structural constraintsof the imaging device 100.

In operation, illumination passes from the light sources 108 throughillumination optics 106 that shape the illumination to a desired form,such as in the form of flood light, light lines, light points, and thelike. The light sources 108 may be narrowband sources such asmonochromatic LED's or laser diodes, or may be broadband sources such aswhite-light LED's or incandescent sources. In cases where the lightsources 108 comprise a series of sources, the series of sources may beof the same wavelength or different wavelengths. The different sources108 may be configured identically or they may differ from each other.

The illumination optics 106 are shown for convenience as consisting of alens but may more generally include any combination of one or morelenses, one or more mirrors, optical windows, and/or other opticalelements. The illumination optics 106 may also comprise a scannermechanism or a spatial light modulator (not shown) to scan theillumination light in a specified one-dimensional or two-dimensionalpattern. The illumination optics may also comprise variable-focuselements such as liquid lenses, deformable mirrors, and other suchdevices to accommodate different object distances. The light source 108may comprise a point source, a line source, an area source, or maycomprise a series of such sources in different embodiments.

After the light passes through the illumination optics 106, it passesthrough the window 102 to illuminate the machine-readable symbol so thatreflected light is directed to the digital imaging system 118, whichcomprises detection optics 114 adapted to focus the light reflected fromthe symbol onto the array. For example, the detection optics 114 maycomprise a lens (including liquid lenses and other variable-focuslenses), a mirror (including deformable mirrors and other variable-focusmirrors), a pinhole, or a combination of such optical elements or otheroptical elements known to those of skill in the art. In someembodiments, the detection optics 114 may provide for variable positionsof the focal plane using auto-focus methods known in the art.

Both the illumination systems 110 and the digital imaging system 118 mayadditionally comprise optical polarizers 104 and 112. The polarizers 104and 112 may be linear or circular, or a combination of the two.

The digital imaging system 118 may also comprise a color filter array116, which may in some instances be incorporated as part of the camera120. The color filter array 116 may conveniently comprise ared-green-blue filter array in the well-known Bayer pattern or in otherpatters. In some instances, the filter elements may function to transmitwavelengths that differ from the standard red-green-blue wavelengths,may include additional wavelengths, and/or may be arranged in a patternthat differs from the Bayer pattern.

The imaging-device layout and components may advantageously be selectedto minimize the direct reflection of the illumination into the digitalimaging system 118. In one embodiment, such direct reflections arereduced by relatively orienting the illumination and detection opticssuch that the amount of directly reflected light detected is minimized.For instance, the optical axes of the illumination optics 110 and thedetection optics 118 may be placed at angles such that a mirror placedon the screen 102 does not direct an appreciable amount of illuminationlight into the detection system 118.

The camera 120 may be coupled electronically with elements of acomputational system that aid in processing of images collected by theimaging device 100. In particular, hardware elements of such acomputational system may be electrically coupled via bus 134, and mayinclude a processor 124, a storage device 128, a processing accelerationunit 236 such as a DSP or special-purpose processor, and a memory 240. Acommunications system 114 may additionally be provided in thoseembodiments where the imaging device 100 is equipped for communicationwith a network In embodiments that include a communications system 114,it may comprise a wired, wireless, modem, and/or other type ofinterfacing connection and permits data to be exchanged with thenetwork.

Software elements are shown as being currently located within workingmemory 140, including an operating system 124 and other code 148, suchas a program designed to implement methods of the invention. It will beapparent to those skilled in the art that substantial variations may beused in accordance with specific requirements. For example, customizedhardware might also be used and/or particular elements might beimplemented in hardware, software (including portable software, such asapplets), or both. Further, connection to other computing devices suchas network input/output devices may be employed.

In particular embodiments, the generic structure shown in FIG. 1 takesthe form of an optically multiplexed multi-imaging sensor such asillustrated in FIG. 2. For example, such a sensor 200 can include aplurality of different illuminators 204 that illuminate a symbol 212with different illumination spectra. A color imager 208 can be used tosimultaneously collect images of the symbol while being illuminated withthe different illuminators. The color imager 208 may comprise a colorfilter array such as a Bayer color filter array that separates theincident light into red, green, and blue components. The threeilluminators 204 emit light with three distinctly different illuminationspectra. In general, these three spectra may be different than the passbands of the color filter array, and some or all of the spectra maysignificantly affect two or more of the color channels. For example, theilluminators 204 might appear to be white, orange, and teal. Also withinthe scope of the invention, any or all of the illuminators 204 maycontain wavelengths in the ultraviolet and/or infrared bands that arealso detectable by the imager 208.

As shown in FIG. 2, these illuminators 204 with distinct illuminationspectra can be positioned at different azimuth angles about the imager208 or symbol 212 and/or at different elevation angles. One or more ofthe illuminators 204 may also be a diffuse illuminator, a ringilluminator, a daylight illuminator, a dark-field illuminator, or otherillumination geometries of the sort. The imager 208 can include a colorfilter array that allows it to collect different images of the symbol212 simultaneously. The imager 208 may comprise a polarizer in one ormore of the color channels. For example, the imager 208 may incorporatea linear polarizer or a circular polarizer in one or more colorchannels. One or more illuminators 204 may also incorporate a polarizer(linear, circular, etc.).

In some embodiments, a sensor can acquire a single image with threedifferent illumination conditions: two directional illuminators withdifferent azimuth angles and a single diffuse illuminator. Eachilluminator in such an embodiment can provide light with different anddistinct wavelengths or wavelength bands. When the machine-readablesymbol comprises a printed barcode, the different illumination spectramay provide enhanced information content in order to detect and/ordecode the barcode, but the configuration may be used for imaging of anymachine-readable symbol, including direct-part marks.

Different illumination variations may be directional or diffuse and mayvary in azimuth and/or elevation angles. Optical multiplexing methodscan be used to multiplex polarization and other optical variationsinstead of, or in addition to, illumination geometry. For example, someof the illuminators may have linear or circular polarizers incorporatedin the optical path. Some or all of the imager channels may have linearor circular polarizers incorporated in the optical path. In some cases,the optical axis of the polarizer for a particular illuminator may beoriented parallel to the polarizer in a particular imaging channel. Inother cases, the optical axis of the polarizer for a particularilluminator may be oriented perpendicular to the polarizer in aparticular imaging channel. Alternative color filter arrays such asfour-color arrays (e.g., CYMW), vertical color filters, and the like,may be used instead of or in addition to Bayer color filter arrays.Alternatively, multiple monochromatic imagers combined with chromaticbeam splitters may be used. An example of this is provided in FIG. 3,which shows a machine-readable symbol in the form of a two-dimensionalbarcode, and a series of beamsplitters 304. A first image results frominteraction with the first beamsplitter 304-1, which redirects lighthaving a wavelength λ less than a first threshold wavelength w₁; asecond image results from interaction with the second beamsplitter304-2, which redirects light having a wavelength λ less than a secondthreshold wavelength w₂; a third image results from interaction with thethird beamsplitter 304-3, which redirects light having a wavelength λless than a third threshold wavelength w₃; and a fourth image resultsfrom the remainder of the beam, which is left with wavelengths greaterthan the third threshold wavelength W₃.

In some embodiments, two or more illumination sources may be used thatare different from each other and that affect the two or more colorchannels of the imager differently. This is illustrated in FIG. 4, whichis generally similar to FIG. 2, having three illuminators 404 and animager 408 used to illuminate and image a machine-readable symbol 412,but is produced separately to emphasize the different illuminationcharacteristics. For example, the illuminators 404 in this embodimentmay comprise broadband light, such as white light or broad-spectrumcolored light. Broadband illumination may be generated by, for example,using a white-light LED or an incandescent source of some kind. Thebroadband source(s) may be filtered in some way using optical bandpassfilters, shortpass filters, longpass filters, and the like. In this way,two broadband illuminators with measurably different illuminationspectra may be used to illuminate the machine-readable symbol 412 undertwo different illumination geometries. The symbol 412 so illuminated maythen be imaged by a color imager 408, resulting in an opticallymultiplexed image.

In another embodiment, the illumination spectra may comprise a pluralityof discrete illumination wavelengths. For example, each illuminatormight comprise a red-green-blue triplet. The electrical current suppliedto each of the LED's in the triplet may then be adjusted to achievedifferent relative intensities of red, green, and blue light such thatthe mixture appears to have different colors in according with the knownprinciples of color addition. In accordance with the invention, each ofthe illuminators corresponding to a different illumination geometry maybe set to a measurably different apparent color by supplying differentelectrical currents to each of the red, green, and blue LEDs in thetriplet. In this case, even though each of the illuminators comprisesthe same three wavelengths, by changing the ratio of the wavelengthintensities, each triplet can affect the channels of the color imagerdifferently.

In a further embodiment, each of the illuminators corresponding todifferent illumination geometries may be nominally monochromatic anddistinct as illustrated generally by FIG. 2. It is known in the art thatthe passbands of red, green, and blue elements of a typical Bayer colorfilter array are broad and overlapping as illustrated in FIG. 5. In thecase where red, green, and blue LEDs are used for illumination, asindicated by the black arrows in the graph in the right panel of thedrawing, it can be seen that each of the monochromatic LEDs willsignificantly affect two or more color channels. In this particularillustration, for instance, the blue LED will affect the blue and greencolor channels; the green LED will affect all three color channels; andthe red LED will affect all three color channels. In accordance with thepresent invention, all three LEDs may be used since they each affect thethree color channels differently. This is in contrast to suggestions inthe prior art that it is desirable to omit green LEDs to avoid anyspectral overlap, and to limit the illumination to using only blue andred LEDs.

Advantageously, and in accordance with the invention, the wavelength ofa monochromatic LED or other source does not need to correspond to thepeak wavelength of a particular color channel. The LEDs for opticalmultiplexing may be chosen according to such other convenient criteriaas availability, pricing, brightness, and other considerations, providedonly that the chosen wavelengths affect the multiple color channels inmeasurably different ways than the other selected LEDs.

b. Handheld Imagers

The devices described above for acquisition of images ofmachine-readable symbols are generally provided as fixed devices, withdifferent imaging conditions being effected through the use of fixedilluminators and imagers such that variations in illumination angles,imaging angles, and the like may be realized through appropriate use ofthe fixed illuminators and imagers. In other embodiments,symbol-acquisition devices may instead be provided as nonfixed devices,which may be embodied in the form of a variety of handheld devices thatinclude imaging capabilities. For example, modern mobile telephones andtablet computers include cameras that may be used as imagers, andprocessing may be effected in accordance with the description below byincluding appropriate applications on the handheld device.

For purposes of illustration, FIGS. 6A and 6B provide structural andfunctional schematic views of a typical mobile device 600, with whichusers may interact through a touchscreen 606 from which a number ofapplications 608 may be accessed, as well as through other buttons 612provided on the device. According to embodiments of the invention, oneof the applications 608 may be configured for acquisition of amachine-readable symbol.

The functional illustration of FIG. 6B is also intended to be exemplary.While the illustration identifies a number of specific functionalcomponents, it is to be understood that alternative devices may lacksome of these specific components and may sometimes include othercomponents not specifically described. The illustration of FIG. 6Bincludes a battery 628 coupled with other components of the device 600through a bus 642, with those components that enable interaction betweenthe device 600 and a user identified collectively with reference number620 and include one or more displays 624, one or more touch sensors 632,hardware buttons 636, and a camera 626.

The camera 626 may be operated by a camera module 180 provided inaddition to other modules used in providing other functionality for thedevice, including an audio module 672, a GPS module 684, anaccelerometer module 676, an input/output module 668, an antenna 664,and a communications module 660. The structure of these modules andtheir interaction with the system is drawn intentionally to bereminiscent of the structure shown in FIG. 1, with software elementsbeing located within working memory 640, including the device'soperating system 644 and other code 648 to operate the different modulesand to implement methods of the invention. The various modules andsoftware may have their operation coordinated by a processor 652 thatinteracts with a storage module 656. The processor 652 may be embodiedas one or more application-specific integrated circuits (“ASICs”), oneor more field-programmable gate arrays (“FGPAs”), or one or moregeneral-purpose processors operative to execute machine-readableinstructions in the form of code. Moreover, the processor 652 mayinclude processing acceleration in the form of a digital-signalprocessor (“DSP”) or special-purpose processor.

The handheld device 600 of FIGS. 6A and 6B may be used to acquire imagesof machine-readable symbols using the method outlined by the flowdiagram of FIG. 6C. At block 690, the user positions the handheld devicein a first location so that the camera may be used at block 692 tocollect an image of the machine-readable symbol at block 692. This isrepeated for multiple positions of the handheld device 600, as indicatedby the loop associated with block 694. While this provides a readymethodology for acquisition of images from different imagingorientations, it is noted that the device 600 may also be used toprovide different illumination orientations and may in some instances beused to provide different spectral qualities of illumination by usingthe device's screen 606 as a source of illumination. Good spectralcontrol may be achieved with devices that having appropriateillumination sources as part of the screen 606, with devices that makeuse of LEDs for illumination being capable of providing substantiallymonochromatic illumination, polychromatic illumination, diffuseillumination, and other forms of illumination such as may be used in thevarious processing methodologies described below. Certain embodimentsthat make use of a handheld device 600 also make use of an ability ofthe camera 626 to vary its focus so that images collected at block 692include images with different focuses, and hence differentmagnifications.

It is generally expected that the level of control over the precision ofthe illumination and/or imaging conditions is less with a handhelddevice of FIGS. 6A and 6B than with a fixed device of FIGS. 1-4, butcompensations may be made through image registration at block 696. Avariety of techniques are known in the art for registration throughtransformation of different data sets onto a common coordinate system,thereby compensating for user imprecision in using the handheld device600. Both intensity-based and feature-based image-registration methodsmay be used in different embodiments. Various transformation models mayalso be used for to effect registration, including lineartransformations that apply linear transformations of translation,rotation, scaling, and other affine transformations. In someembodiments, elastic transformations may also be applied for localwarping of the image, including the use of radial basis functions andlarge deformation models.

Once the collected images have been registered, they may be combined orprocessed as described elsewhere herein to read the machine-readablesymbol at block 698.

b. Artifacts

In some circumstances, the machine-readable symbol may be present on asurface that potentially gives rise to certain imaging artifacts. Forexample, a printed barcode may be produced on a glossy substrate such asa plastic of some kind. Alternatively, the symbol may comprise adirect-part mark made in glass, metal, or some other glossy materialusing peening, cutting, casting, chemical etching, laser etching, oranother of such methods.

In these cases, images of the symbol may be obscured in whole or in partby glints, specular reflections, poorly illuminated regions, and otherillumination and/or imaging artifacts. In particular, illuminationvariation of the symbol may cause the image of the symbol to be obscuredin whole or in part. Principles for this aspect of the invention areillustrated for a one-dimensional barcode in connection with FIGS. 7-12,but it will be appreciated that the same principles operate with othertypes of machine-readable symbols, including two-dimensional and colorbarcodes, as well as direct-part marks and others. FIG. 7A shows eightimages of a one-dimensional barcode collected with illumination light ateight different azimuth angles and imaged by a stationary imager. Inthis case, the illumination was at an elevation of approximately 75°(measured from the optical axis of the imager) and azimuth angles of 0°,45°, 90°, 135°, 180°, 225°, 270°, and 315°. The imager was a CMOS imagerand the light sources were LEDs with a central wavelength of 400 nm. Ascan be seen from the drawing, the regions of bright and dark intensitiesdue to the surface topography change according to the illuminationorientation.

A set of such images may be processed in various ways according toembodiments of the invention to reduce or eliminate such illuminationvariations. For example, a pixelwise sort of the eight images wasapplied, the operation resulting in the eight new images shown in FIG.7B, with the images redisplayed as images from low sort values (upperleft) to high sort values (third row, second column). In this way, amore consistent, higher-contrast single image is generated by combiningthe information from the eight raw images. While eight illuminationsources were used in this example, there is no restriction on the numberof illumination sources that may be used in different embodiments.

It is noted that the upper-left image of FIG. 8 corresponds to thepixelwise minimum across the eight raw images as is sometimes done inthe prior art. This minimum-pixel-value barcode is unable to be properlyread by a commercial online decoder. In contrast, some of the othersorted images are capable of being properly read and decoded,specifically the sixth, seventh, and eighth highest values, whichcorrespond to the second row, third column image and the two images inthe third row.

This sorting operation is merely an example of a more general set ofprocessing transformations applied according to embodiments of theinvention. For instance, in another example, a bilateral filter isapplied to one or more of the images (or to a composite of the images)using edge information generated by a different method, such as a medianof gradients. In some cases, it is advantageous to combine all of theimages in some way. Methods for doing so may include generating one ormore factors from eigenanalysis, independent component analysis, tensoranalysis, and the like, or other such operations performed on the rawimages or some derivative of the raw images.

In some instances, a simple averaging operation is sufficient to combinethe raw images and reduce the image artifacts to acceptable levels. Inparticular, in cases where there are a relatively large number of rawimages taken under different illumination conditions and/or theillumination conditions are approximately balanced, averaging may beeffected. “Balanced” illumination refers to a set of illuminationconditions in which the illumination portions of the image vary betweenillumination conditions in complementary fashion. For example, in caseswhere illumination is symmetric in the azimuth angle, i.e. with 180°symmetry, the resulting images display complementary artifacts that maybe reduced through averaging. FIG. 9 shows an example of the benefits ofaveraging with a balanced system. In this drawing, the left image is theresult of averaging the eight images shown in FIG. 7B. The system isbalanced, with light sources at equal elevations (approximately 75°) andseparated by an azimuth angle of 45° in a full circle around the object.In order to demonstrate the benefits of a balanced illumination system,the middle image shows the result of averaging only four of the rawimages: numbers 1, 3, 5, and 7, which are spaced uniformly at 90° andspan azimuth angles of 0-360°. In contrast, the rightmost image resultsfrom averaging four raw images that are unbalanced: numbers 1, 2, 3, and4, which are separated by 45° and span azimuth angles of 0-180°. As canbe seen from the drawing, the result of unbalanced averaging retainsmore illumination artifacts than the average of an equal number ofimages generated from a balanced illumination system.

In some cases, the raw images may be uniformly weighted when combinedthrough averaging or other operations. In other cases, a nonuniformweighting may be applied to the images before the averaging or otheroperations. For instance, the quality of each raw image may beascertained by a variety of criteria that include contrast, noise,barcode quality, barcode detectability, barcode readability, amount ofspecular reflections, illumination variability, and the like, with theresulting quality measures being used to weight the images beingaveraged; high-quality images may be weighted more highly thanlow-quality images.

In some cases, it is desirable to determine estimates of key intrinsiccharacteristics of the object from the raw images and to use suchestimates for processing the machine-readable symbols rather thanattempting to decode the raw images themselves. This is so because themachine-readable images are placed on the object by affecting one ormore intrinsic object characteristics such as reflectance, absorbance(or pseudo-absorbance or reflectance), texture, surface profile, color,chromaticity, and other such parameters. But the raw images are theresult of interactions of the object and symbol with the illuminationsystem as well as the imaging system. The raw images contain arepresentation of the interaction of all these different effects, onlysome of which are intrinsic to the object or symbol. In some cases, itis beneficial to attempt to separate the raw images into intrinsic andextrinsic effects, and then use the derived intrinsic characteristic(s)of the object and symbol for subsequent processing and decoding. Oneexample of this approach is illustrated by processing the raw imagesusing various photometric stereo methods to derive measures of surfacetopography as well as albedo or reflectance. This approach is discussedfurther below.

Another example of an intrinsic object characteristic that may beestimated from the raw images is the reflectance or log(reflectance)(also referred to as the pseudo-absorbance) of the objects. In order toestimate the reflectance, the illumination characteristics for each rawimage should be estimated. If constant or otherwise applicable, suchcharacteristics of the illumination system and/or the imaging system maybe calibrated separately and then applied to the raw images. One methodfor estimating the illumination characteristics from the raw images isprovided by smoothing the raw images, assuming that the illuminationvariation is of lower spatial frequency than the symbol and/or othercharacteristics of the object or imaging system. A smoothing such as awavelet smoothing or another type of smoothing known in the art may beapplied. An example of the results of this operation when a waveletsmoothing is applied to the data of FIG. 7B is shown in FIG. 9.

The estimated illumination profiles in FIG. 9 may be used as denominatorfor estimating the object reflectance as

${{R\left( {x,y} \right)} = \frac{I\left( {x,y} \right)}{{S\left( {x,y} \right)} + N}},$where R(x,y) is the object reflectance, I(x,y) is the raw image, S(x,y)is the estimate of the illumination and imaging variations under certainconditions, and N is a regularization term to avoid numericalinstability due to noise, etc. By using FIG. 9 in combination with FIG.7A in this manner, an estimate of reflectance may be found as shown inFIG. 10. The corresponding absorbance (or pseudo-absorbance orlog-reflectance) images are shown in FIG. 11.

While the pseudo-absorbance images shown in FIG. 11 are notartifact-free, they are more uniform than the correspondingraw-intensity images of FIG. 7A. Moreover, some of the barcode featuresthat were in the poorly illuminated regions of the images are expressedwith higher contrast in the pseudo-absorbance images of FIG. 11. Thefirst (upper left) image for the raw-intensity data in FIG. 7A iscompared directly with the pseudo-absorbance data in FIG. 11 for easycomparison in FIG. 12.

In some cases, the stack of images may be processed by methods such asphotometric stereo. Such a technique allows the separate estimation ofthe reflectance (albedo) and the surface topography. Once the surfacetopography is known, it may be used to evaluate points of specularityand correct for them in a variety of ways. The surface topography mayalso be used to correct for the nonplanarity of the symbol in thosecases where it is a nominally planar symbol, such as in the case of aprinted barcode. Alternatively, if the symbol is a direct-part markmachined or etched into the substrate, the surface topography may be abetter representation of the symbol than any of the raw images or thederived composite reflectance image.

In embodiments where the illumination comprises white-light or otherbroadband illumination, the chromatic characteristics of the object(symbol and background) may provide a relevant source of informationabout the symbol pattern and/or interfering specularities. For example,specularities have a chromatic content that is closer to that of theillumination source rather than those of the object being illuminated.As such, the chromaticity of the raw images may be analyzed to determinethose pixels or groups of pixels with chromatic content that is close tothat of the light source. Those pixels may be excluded or down-weightedwhen the plurality of raw images are combined in some way to produce acomposite symbol image.

c. Chromatic Multiplexing

In some embodiments, it is desirable to have light sources withdifferent characteristics illuminate a machine-readable symbol. Suchcharacteristics may include wavelength, polarization, azimuth angle,elevation angle, directional versus diffuse illumination, and the like.In some cases, multiple images of a machine-readable symbol may becollected by a single imager under multiple illumination conditions bycollecting a sequence of images: for example, one of a plurality ofsources could be illuminated, an image acquired, the source extinguishedand a second source illuminated, a second image acquired, and so on.

In other cases, images are collected for multiple illuminationconditions simultaneously. In particular, embodiments of the inventionallow for acquisition of machine-readable-symbol images from multipleillumination geometries simultaneously. For example, a symbol might beilluminated with red light oriented at a certain azimuth and elevationangle; with green light oriented orthogonally to the red light in theazimuth direction and either the same or different elevation angle; andwith blue light that diffusely illuminates the object, with all suchilluminators turned on simultaneously. Such a configuration may then beimaged by a color imager that comprises a Bayer imager with red, green,and blue pixels arranged as a color filter array. The resulting rawimage may then be separated into three images corresponding to each ofthe red, green, and blue color channels. Each of these channelscorresponds to a different illumination geometry as well as to adifferent wavelength. The simultaneous measurement of differentillumination conditions is referred to herein as “chromaticmultiplexing.” The three resulting images may be analyzed separately orcombined in some way to extract an image of the machine-readable symbolthat may then be processed further.

Similar chromatic multiplexing may be performed using more colors orfewer colors or different colors, according to characteristics of thecolor imager. In one embodiment, the illumination colors substantiallycorrespond to the passbands of the individual filter elements within thecolor imager. Also, each such multiple colors of illumination mayalternatively or additionally differ by the orientation of polarization(or the absence thereof) relative to a polarizer present in the imagingsystem and/or relative to the polarization characteristics of the symbolor other components.

Alternatively, multiple noncolor imagers may be used in conjunction withchromatic beamsplitters to accomplish a similar separation of thechromatically multiplexed images as was performed using a color imagerthat incorporates a color filter array. An example of this configurationis shown in FIG. 3 and described elsewhere in this application. As well,other mechanisms of recording and separating different color channelsmay be used.

3. Symbol Types

a. Dual-Mode Machine-Readable Symbols

Certain embodiments of the invention make use of multiple aspects ofconventional machine-readable symbols to produce a dual-mode ormulti-mode symbol. In one exemplary embodiment, a dual-mode symbol isfabricated by using both the reflectance and topography to instantiateseparate patterns. One pattern may be printed on a surface, which isreferred to herein as a “reflectance barcode” and a different patternmay be incorporated in the surface topography before, during, or afterthe reflectance barcode, in substantially the same or overlappinglocation as the reflectance barcode. This second pattern is referred toherein as a “topographic barcode.” The result is a composite of twodifferent patterns manifested using two different characteristics of anobject.

This is illustrated in FIG. 13 where the reflectance barcode 1304 isprinted on a thin, conformal membrane and the topographic barcode 1308is produced by machining, laser-etching, or similar technique into asubstrate that may be made of metal, plastic, or some other material.The conformal layer is applied over the substrate with the reflectanceand topographic barcodes 1304 and 1308 overlapping to produce thedual-mode symbol 1312. Similar dual-mode symbols can be generated byprinting a first barcode on a deformable material such as paper or sheetmetal. A second barcode may then be impressed on some or all of thefirst barcode using embossing or stamping. In another embodiment, atopographic barcode may be machined into the substrate using standardmethods. A second barcode may then be painted on the first barcode usingspraying and other methods known in the art.

The resulting dual-mode symbol 1312 may, for example, be imaged using asingle imager and a variety of illumination angles. Such a set of imagesmay then be processed using photometric-stereo processing techniques toseparately evaluate the reflectance and topographic characteristics thatarise from the reflectance barcode 1304 and topographic barcode 1308respectively. The separately evaluated images may then be decoded in theusual way to extract messages from each of the barcodes so formed.

Other types of multi-mode symbols also fall within the scope of theinvention. For example, barcodes may be combined using properties suchas polarization characteristics, spectral characteristics in the form ofcolor, textural characteristics, and the like. In this way, two or moremachine-readable symbols instantiated using different characteristicsmay be combined together and later separated by collecting andprocessing multiple different images of the symbol region.

Alternatively, different characteristics may be combined together toprovide a symbology of a single barcode that is of higher order thanbinary. For example, by combining reflectance and depth, the symbolscould be white-surface, white-deep, black-surface, black-deep, leadingto a quaternary symbol set. Other combinations such asreflectance-texture, reflectance-polarization, depth-texture,depth-texture-reflectance, and so on may be combined within the scope ofthe invention.

b. Direct-Part Marks

As previously noted, direct-part marks may advantageously beincorporated as part of an item by modification of a surface of the itemusing such techniques as laser etching, chemical etching, dot peening,casting, machining, and the like. The optically multiplexedmulti-imaging sensors described in connection with FIGS. 1-5 may be usedfor acquisition of such direct-part marks, taking advantage of themulti-imaging capabilities of the sensors.

Such a sensor was used by the inventors to collect images from a varietyof different data-matrix direct-part marks. One example of a set of red,green, and blue images collected in such a manner is shown in FIG. 14for a machine-readable symbol created by using a dot-peening process sothat the symbol is manifested through small depressions in a metalsubstrate. The sensor in this illustrated used green and blue sourcesthat were provided as directional LED's while the red source was anon-axis diffuse light source, and the imager configured to view theobject coaxially to the red diffuse source via an optical beamsplitter.The left portion of the drawing shows the red color plane, the centralportion of the drawing shows the green color plane, and the rightportion of the drawing shows the blue color plane. As obvious from thedrawing, the images of the symbol collected under direct illumination(central green and right blue) are distinctly different than the imagesacquired with diffuse (left red) illumination. These three images may beviewed as a single color image as shown in FIG. 15.

The presence, position, and orientation of this symbol may be determinedusing methods known in the art, and the raw images may be preprocessedbefore or after the detection is performed. As an example of a detectionmethod, a DataMatrix barcode such as illustrated in FIGS. 14 and 15 isknown to have two “solid” edges and two edges with alternating on/offelements. Such prior knowledge facilitates the use of image-processingtechniques such as line detectors, edge detectors, corner detectors, andthe like to be applied to the images to determine the boundaries of thebarcode. Preprocessing of the barcode may include noise filtering,correction for nonuniform illumination, geometric correction, contrastenhancement, spatial filtering, and other processing of the sort. Oncethe symbol is detected and located, the boundary and the individualcells of the symbol may be determined as illustrated in FIG. 16, withlines drawn in the figure to define a grid in which individual cells areeither peened or not peened.

A close-up of a small number of cells in FIG. 17 shows the nature of thethree different images collected under the three illuminationconditions. As readily observed, the red diffuse illumination causes thesubstrate metal to appear as bright, except in regions in which thedot-peen depression is present, which instead show up as dark circles.In contrast, the direct green and blue illumination are dark everywhereexcept in the depressions and in other imperfections in the metalsubstrate. Furthermore, the location of the bright (specular) spot dueto the green illuminator is in a different location from the bright spotdue to the blue illumination, as expected from the geometricconsiderations associated with specular reflections as described above.

Because each of the raw color images manifests the features of thesymbol differently, these images are not well-suited to being directlycombined through image fusion and other such techniques. Instead,embodiments of the invention directly analyze the spatio-spectralcharacteristics of each cell to determine to which of the binary states(peened or not peened) the cell corresponds.

Once the barcode position and orientation are determine, there are anumber of reference features of the data-matrix symbol that may be usedto determine examples of the “on” and “off” states of this particularsymbol collected under this particular set of illumination conditions.For example, the data-matrix symbol has two solid edges that may be usedfor locating and orienting the symbol, two edges with alternating on-offpatterns that are used to determine the number of cells, a quiet zonethat surrounds the symbol with “off” cells, etc. Other direct-partsymbols have similar portions that are defined by applicable standards.

Such reference portions of the symbol may be used as reference cells todetermine the optical characteristics of the two states of the symbol.FIG. 18 shows the cells that were automatically selected as referencecells. The reference cells with the dot peen are marked with blue andthe reference cells without a dot peen are marked in red. In thisparticular case, only the location edges (right and lower) and thetiming edges (left and top) were used. Additional information could havebeen acquired from the quiet zone that surrounds the symbol. In the caseof larger data-matrix symbols, as well as in other types of symbols,there may also be reference structures present in the intermediateregion of the symbol, but in this particular case the cells inside thecolored edges are unknown states that are determined by the methods ofthe invention.

The statistical properties of the cells (reference and unknown) may beassessed and summarized in a variety of ways in different embodiments.For example, histograms of the pixel values within each cell for each ofthe color planes may be generated. The mean, variance, and higher-orderstatistics of the pixel values for each of the pixels within the cellsand each of the color planes may be determined. Spatial descriptors ofeach cell contents may be summarized with power spectral densities,discrete cosine transforms, wavelets, and other methods of the sort. Avariety of textural measures of the cell may be made.Principal-component analysis of the cell contents in the spatial,spectral, or spatio-spectral dimensions may be performed and the resultscompiled. All of these characterizations may be performed on the rawcolor data or in alternative color spaces such as L-a-b, YCbCr, etc.

Whatever characterization of the cells are made, these values may beused as features for a subsequent classification step. In the case ofthe present example, each cell was summarized by the mean and thestandard deviation of the values for all pixels within the cell for eachof the red, green, and blue color planes. The same summary values werealso generated for the corresponding YCbCr data. These twelve values percell were then used as features for a subsequent classification usinglinear discriminant analysis (LDA), which is merely one example ofclassification methods that may be used. Other embodiments, forinstance, make use of nonlinear discriminant analysis, AdaBoost, neuralnetworks, K nearest neighbors, and a variety of other methods.

The features taken from the reference cells may be used to set theparameters of the classifier and to optimize the performance for thatsymbol. Once the classifier is set or trained, it may then be applied tothe unknown cells of the same image, as well as for other such similarimages. The result of applying the classifier to unknown cells of thedot-peen symbol is shown in FIG. 19. As can be seen by reviewing theclassification results, which used red/blue coloring, as compared to theunderlying grayscale image, the results of the classification were 100%accurate.

Once the classification results for each cell are determined, theresults may be compiled into a simple binary grid, which may then besubsequently decoded. An additional benefit of the present method isthat any existing geometric distortion of the raw data does not need tobe explicitly corrected. Rather, the statistical properties of thepixels within a cell may be used to determine the state of the cell,which is then recorded in an appropriate square or rectangular grid (orwhatever shape corresponds to the ideal form factor for the symbol underanalysis). In so doing, the aspect ratio is removed from the finalresult without requiring a computationally expensive image-processingprocedure.

A second example of the methods of the invention is provided with asymbol having poorly formed elements generated from spot welds, as shownin FIGS. 20-22. The same sensor was used for this analysis as describedabove in connection FIG. 14. Similar to FIG. 14, the left portion ofFIG. 20 shows the red color plane (obtained using diffuse redillumination), the central portion of FIG. 20 shows the green colorplane (obtained using direct green illumination), and the right portionof FIG. 20 shows the blue color plane (obtained using direct blueillumination). Similar to FIG. 15, the three images may be viewed as asingle image in FIG. 21. The close-up of the central portion of thesymbol shown in FIG. 22 clearly illustrated the poorly formed elements(in the lower right of the drawing) as well as damage to other elements(in the lower left quadrant of FIG. 22).

The same procedure for statistical interpretation of the symbol patternas described above was applied also to this symbol. In this case, thereference cells are shown color-coded in FIG. 23, with results of theclassification shown in FIG. 24. The classification results are againcorrect, and the resulting binary image decoded without errors.

c. Copy-Resistant Symbols

Embodiments of the invention are also directed to machine-readablesymbols that are copy-resistant. Such copy-resistance may be effected byincorporating an optically variable material, which may be comprised bysymbol itself, by the substrate on or in which the symbol isinstantiated, or by a supplementary layer. Examples of supplementarylayers that may be provided include a laminate, coating, or othercovering layer that is generally coextensive with the symbol, althoughlayers that are not coextensive may also be used in some embodiments.

In addition, at least one of the machine-readable symbol, the substrate,and the supplementary layer (if present) comprises a security feature,defined above as referring to an identifying marking that takes the formof words, symbols, patterns, colors, textures, fiducial marks, surfacefinishes, and the like.

Methods of rendering the machine-readable symbol copy-resistant areillustrated with the flow diagram of FIG. 25. While the flow diagramsets forth a number of steps explicitly and provides an illustration ofthose steps in a particular order, neither of these is intended to belimiting; in alternative embodiments, some of the steps may be omitted,additional steps not specifically called out may be performed, and/orsome of the steps may be performed in an order that is different fromthat illustrated. In discussing FIG. 25 below, comments are also madewith respect to FIG. 26, which provides a schematic illustration of astructure for a copy-resistant symbol. While the example is shown in theform of a two-dimensional barcode, it will be understood by those ofskill in the art that the same principles may be applied to other typesof machine-readable symbols discussed herein.

According to embodiments of the invention, the machine-readable symbolincludes both substantive information and security information, whichare collectively encoded into the machine-readable format according toencoding schemes that are readily available. The security information,though, is further encrypted prior to that encoding according to aseparate encryption scheme that is preferably “secret” in the sense thatit is known only by a limited number of parties who have been authorizedto have knowledge of that encryption scheme. Such encryption may use anyof a variety of techniques, including both symmetric and asymmetricschemes. For example, the authorized parties may have access to adecryption key that allows for the security-information component to bedecrypted. The security information itself includes some specificationof the security feature incorporated by the copy-resistant symbol,allowing verification of the presence of the correct security feature.

Thus, in order to read a machine-readable symbol as outlined by theexemplary method of FIG. 25—and to verify that it is not presented aspart of a copy of an otherwise legitimate symbol—the symbol is presentedto the optical reader at block 2504. At block 2508, an initial check ismade whether the symbol includes optically variable characteristics,with the symbol being rejected at block 2512 if it does not. Thisprovides a high-level verification that the symbol at least has therequired structure and, in accordance with the structure previouslydiscussed, the optically variable characteristics may be present in thesubstrate of the symbol, in the machine-readable symbol itself, or in asupplementary layer. In some embodiments, more specific properties ofthe optically variable characteristics may be required in order toproceed. For instance, a symbol may be considered invalid if it lacks aspecific holographic signature, the lack of such a signature suggestingthat it may be a nonholographic copy of a genuine symbol.

If optically variable characteristics are present, the machine-readablesymbol is read by the optical reader at block 2516. Decoding of themachine-readable symbol at block 2520 proceeds by applying theappropriate algorithm in accordance with the particular symbology usedto encode the symbol. The effect of this step is illustrated in FIG. 26with a two-dimensional barcode 2600 that has been decoded to render analphanumeric string that has both substantive information 2604 andencrypted security information 2608. An attempt is accordingly made toparse the resulting alphanumeric string so that the security informationcan be decrypted.

At block 2524, the decryption key is obtained. This may be done in anumber of different ways in different embodiments. In some cases wherethe security information 2608 has been encrypted using asymmetric orpublic-key encryption, the purportedly proper decryption might bemaintained in a database that is either publicly or privately available.When it is maintained in a publicly accessible database, thecommunications system 132 of the optical reader may be used to connectto a network and access the database to retrieve the decryption key.This might be a suitable arrangement, for example, where a manufacturerpurchases critical components from a supplier and wishes to authenticatea particular shipment of components by retrieving the vendor's publickey as published in a publicly accessible database so that it may thenbe applied to the message extracted from a symbol present on ajust-received shipment. When the key is maintained in a privatedatabase, it may be accessible directly to the optical reader by beingstored on the storage device 128.

While such a mechanism of maintaining a key is possible with asymmetricencryption, it is expected to be more usual in those embodiments inwhich symmetric encryption is used. In such embodiments, the common keymay be shared between entities using some secure mechanism. In theexample of a vendor shipping components to a manufacturer, the twoparties might agree to use a particular encryption key for transactionbetween the two entities. This may advantageously be effected by using akey-sharing algorithm to transmit the key in a secure fashion. Oneexample of such a key-sharing algorithm is the Diffie-Hellman algorithm,in which case some or all of the public elements of the Diffie-Hellmankey sharing may be included as an unencrypted part of the encodedmachine-readable-symbol message.

Application of the decryption key to the security portion 2608 of thedecoded message at block 2528 of FIG. 25 is illustrated in FIG. 26 toresolve the unencrypted message. The unencrypted security message allowsfor a specification of the security feature of the symbol to bedetermined at block 2532 of FIG. 25. This may result from any of avariety of formats, one of which is suggested in FIG. 26 forillustrative purposes. In this example, the format specifies that thesecurity feature is the presence of the word “AUTHENTIC” 2612 as part ofthe optically variable component of the symbol and that the word isinclined at an angle of 45° across the machine-readable symbol asindicated by the resolved relationship information 2616.

Successful decryption of the security information 2608 allows acomparison to be made at block 2536 of FIG. 25 between the featurevalues specified in the security information with an actual securityfeature present on the symbol. If the decrypted feature informationmatches the feature actually present on the symbol, then there isgreater confidence that the symbol was produced by an authorized entityand action can be taken at block 2540 with the substantive informationcontained within the machine-readable symbol. Conversely, if thedecryption fails to yield intelligible feature values or if thedecrypted feature values do not match the actual features of the symbolfrom which they were extracted, this an indication that the symbol waseither not produced by an authorized entity or was altered in some way,prompting the symbol to be rejected at block 2512.

There are numerous features that may be used as security features indifferent embodiments. Indeed, the availability of such numerousfeatures increases the resistance to copying since it is not apparent tounauthorized parties which feature is the actual security feature. FIG.27A shows an example of a symbol substrate that is holographic. In thisexample, this substrate has the words “AUTHENTIC” and “ORIGINAL” printedon it, but more generally a holographic substrate may have any words,symbols, patterns, scenes, or other marks printed on it, only a subsetof which might be the true security features.

FIG. 27B shows an example of a machine-readable symbol in the form of atwo-dimensional barcode printed over the substrate of FIG. 27A. In thisexample, the barcode was printed with a thermal heat-transfer process,but a variety of other printing processes could alternatively be used.

FIG. 28 illustrates the converse of an opaque barcode printed over aholographic substrate by showing an example where the substrate isnonholographic and the barcode itself is holographic. In this example,the barcode material is a holographic heat-transfer material that isavailable commercially. This example also illustrates the use of apattern of dots as a security feature rather than the presence of wordsin the holograms of FIGS. 27A and 27B. The size, shape, spacing, and/orcolor of the dots are examples of characteristics that may be used as asecurity feature, with information regarding such characteristicsprovided as part of the security information encoded within the barcodeas described above.

The substrate in this example is, moreover, substantially uniform incolor with no explicit printing of words, symbols, textures, or othermarks. In other embodiments, the substrate could additionally containsuch words, symbols, colors, textures, patterns, or other marks, some orall of which could also be used as part of the security feature. Instill other embodiments, both the machine-readable symbol and thesubstrate could be holographic with differing characteristics.

Alternatively or in addition to describing the security feature itself,a feature type or category may be used. For example, the opaque barcodeof FIG. 27B is generally white in color and may be designated as “Type37” while a similar barcode produced with read opaque heat-transferribbon could be designated as “Type 38” and the barcode illustrated inFIG. 28 could be designated as “Type 67.” In this way, multipledifferent characteristics of different instantiations ofmachine-readable symbols may be grouped into a single type designator,with available databases then being used to look up specific featuresand/or images of representative instantiations of a certain type.

In addition, spatial relationships between different elements of thesymbol may be used as features. This is illustrated in FIGS. 29A and 29Bfor symbols of the type shown in FIG. 27B, i.e. for an opaque whitetwo-dimensional barcode printed on a holographic substrate. In FIG. 29A,the angle θ of the marks on the substrate relative to a reference linedefined by the machine-readable symbol may be used as a feature. Such afeature was used as part of the example in FIG. 26 above, in which theword “AUTHENTIC” needed to be inclined at an angle of 45° in order to bevalid as a security feature.

Similarly, the displacement of some point on the machine-readable symbolrelative to some mark on the substrate may be used as part of thesecurity feature. This is illustrated in FIG. 29B for a Cartesiandisplacement (X, Y). Any such spatial relationship between differentelements of the symbol may be used as features, both between differentcomponents of the symbol—i.e. between points on the machine-readablesymbol and substrate, between points on the machine-readable symbol andsupplementary layer, or between points on the substrate andsupplementary layer. Fiducial or registration marks may accordingly beincluded within the symbol to facilitate such feature measurements.

To increase the robustness of such spatial measurements, it may beadvantageous to maintain relatively large step sizes between differentpossible configurations, although the appropriate size steps may dependon both manufacturing capabilities and on resolution capabilities ofoptical readers to be used in scanning the symbols. Merely by way ofexample, angular measurements may be limited to possible angles of 0°,45°, 90°, 135°, 180°, 225°, 270°, and 315°, which would be relative easyto determine and to distinguish. In a similar manner, the lineardisplacement between reference points might be coarsely described bywhich octant a fiducial mark is to be found in relative to themachine-readable symbol or other identifying position using compassdirections: N, NW, W, SW, S, SE, E, or NE. Alternatively, thedisplacement could be described in terms of physical units (e.g. inches,mm, etc.) using Cartesian, polar, or other coordinate systems. In anycase, the linear and angular displacement step sizes are preferablechosen to be unambiguously measured and validated in order for thesefeatures to be most effective.

Having described several embodiments, it will be recognized by those ofskill in the art that various modifications, alternative constructions,and equivalents may be used without departing from the spirit of theinvention. Accordingly, the above description should not be taken aslimiting the scope of the invention, which is defined in the followingclaims.

What is claimed is:
 1. An optical reader comprising: an illuminationsource disposed to illuminate a presented copy-resistant symbolincluding machine-readable symbol; a detection device disposed toreceive light scattered from the presented copy-resistant symbol; and acomputational unit interfaced with the detection device and having:instructions to optically acquire the machine-readable symbol from thepresented copy-resistant symbol; instructions to first decode themachine-readable symbol to derive a message; instructions to seconddecode a portion of the message by applying a decoding process to theportion of the message; instructions to determine a security featurefrom the decoded portion of the message; and instructions to opticallyconfirm a physical presence of the security feature on the presentedcopy-resistant symbol with the detection device.
 2. The optical readerrecited in claim 1 wherein: the portion of the message is encryptedaccording to an asymmetric encryption algorithm; and the instructions todecode the portion of the message comprise instructions to apply apublicly accessible decryption key to the portion of the message.
 3. Theoptical reader recited in claim 1 wherein: the portion of the message isencrypted according to a symmetric encryption algorithm; and theinstructions to decode the portion of the message comprise instructionsto apply a secure decryption key to the portion of the message.
 4. Theoptical reader recited in claim 1 wherein the computational unit furtherhas instructions to confirm that the presented copy-resistant symbolcomprises an optically variable material.
 5. The optical reader recitedin claim 1 wherein the machine-readable symbol comprises a barcode. 6.The optical reader recited in claim 1 wherein the presentedcopy-resistant symbol further comprises a substrate having themachine-readable symbol printed thereon.
 7. The optical reader recitedin claim 1 wherein the presented copy-resistant symbol comprises asubstrate having a machine-readable symbol incorporated therein.
 8. Theoptical reader recited in claim 1 wherein the presented copy-resistantsymbol further comprises an identifying mark.
 9. The optical readerrecited in claim 8 wherein the security feature comprises an angularrelationship of the identifying mark and a reference comprised by thecopy-resistant symbol.
 10. The optical reader recited in claim 8 whereinthe security feature comprises a spatial relationship between theidentifying mark and a reference comprised by the copy-resistant symbol.11. The optical reader recited in claim 1 wherein the computational unitfurther has instructions to parse a second portion of the message toidentify substantive information.
 12. The optical reader recited inclaim 1 wherein: the copy-resistant symbol is on an item; and thesubstantive information identifies the item.
 13. The optical readerrecited in claim 12 wherein the security feature comprises a mark madeon the item during manufacture of the item.
 14. The optical readerrecited in claim 13 wherein the mark resulted from a substantiallyrandom outcome of the manufacture of the item.